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Meeting Summary 


The following summary is organized into two major sections. The first section captures the 
individual contribution of meeting participants, including both IES staff and the invited 
technical working group members. The second section captures themes that arose during the 
group discussions, which were organized according to the IES goal structure. The 
observations and suggestions throughout the day were often overlapping and 
complementary. 


Welcome 
Thomas Brock, Ph.D., Commissioner, NCER, and Joan McLaughlin, Ph.D., 
Commissioner, NCSER 


Dr. Brock called the meeting to order at 8:32 a.m. and welcomed the technical working 
group (WG) participants. He and Dr. McLaughlin thanked IES staff members Corinne 
Alfeld and Meredith Larson in particular for organizing this meeting. 


Dr. McLaughlin said that, although many IES-funded researchers do a good job of 
implementing mixed methods research' (MMR), many researchers could use more guidance. 
Feedback from practitioners, researchers, and the public indicates that many people believe 
IES does not support MMR. The goal of this meeting was to gather input both on technical 
assistance and resources that NCER and NCSER could provide to encourage MMR and to 
ensute that it is consistently done in a high-quality manner. In addition, the objectives of the 
meeting (as outlined in a letter to TWG members prior to the meeting; see Appendix A) 
sought to gather suggestions on ways to incorporate information about conducting MMR 
into resources routinely provided to researchers, in grant opportunities, and through 
research training programs. 


Introduction to/Reminder of IES Goal Structure 
Corinne Alfeld, Research Analyst, NCER 


To help frame the day’s discussion, Dr. Alfeld outlined the five IES goals around which the 
bulk of IES grant-driven research funding is organized: 


e Goal 1, Exploration, supports research that seeks to identify malleable factors 
associated with student education and the factors or conditions that mediate or 
moderate that relationship. 

e Goal 2, Development and Innovation, supports the development of new 
interventions or improvements to existing interventions. Such research often 
involves an iterative process and a pilot study. 

e Goal 3, Efficacy and Replication, seeks to examine the impact of interventions on 
student outcomes relative to a counterfactual. As such, it supports rigorous research, 
but interventions can be implemented under ideal (rather than actual) conditions. 


'For the purpose of this meeting, the term “mixed methods research” referred to combining 
qualitative and quantitative research. 


Mixed Methods in Education Research IES TWG | Meeting Summary 3 


e Goal 4, Effectiveness, examines the impact of interventions under routine practice 
conditions, is conducted by evaluators independent of the intervention’s developer, 
and includes assessment of the fidelity of implementation. 

e Goal 5, Measurement, supports the development of new assessments or 
measurements or the validation of existing assessment tools. 


Appendix B provides a more in depth description of the research goal structure used by 
NCER and NCSER. 


Prior to the meeting, participants were asked to prepare brief descriptions of their MMR 
projects under of the research goals, including lessons learned and issues for discussion, as 
the basis for conversation. The summaries below capture these presentations in the order in 
which they occurred, while the observations and suggestions from the subsequent group 
discussion are captured and synthesized at the end of the document. 


Mixed Methods in Exploratory Research (Goal 1) 
Peggy Estrada, Ruby Mendenhall, and Mario Small 


Peggy Estrada, Ph.D., Associate Research Scientist, Latin American and 
Latino Studies Department, University of California—Santa Cruz 


Dr. Estrada described her research on English language learners (ELLs) in California and the 
state’s reclassification of students to “fluent English-proficient” status. She and her 
colleagues focused on the reclassification process; factors that impede or facilitate 
reclassification; characteristics of those who are and are not reclassified; access to cote 
content; and the relation of reclassification to academic achievement. Dr. Estrada said she 
and the other investigators aimed to identify patterns, understand how those patterns 
emerged, and discover causal relations in the data. 


Findings from her previous qualitative research led Dr. Estrada to hypothesize that some 
students meet criteria but do not get reclassified and that ELLs, particularly at the secondary 
level, have less access to core content. Testing these hypotheses and understanding the 
complex phenomena around reclassification and access required a large-scale study. The 
current study uses quantitative research to find patterns and qualitative research to elucidate 
them. 


Dr. Estrada said her work demonstrates the potential of MMR to produce deep knowledge. 
For example, quantitative analysis of student data and forms showed a pattern in which a 
substantial number of students met criteria but were denied reclassification. Reclassification 
forms showed teacher denials. Interviews with school staff revealed teacher rationales for 
denials. Additionally those data showed that when the reclassification process was labor- 
intensive, provided tacit authority to teachers to approve or deny reclassification, and staff 
knowledge was inadequate, denials were much mote likely. 


Dr. Estrada emphasized that the research team would not have been able to identify the 
reasons for denied reclassification without an iterative, MMR process that allowed the team 
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to develop new hypotheses and change protocols throughout. However, such an approach is 
conceptually complex and labor-intensive, she cautioned. 


Ruby Mendenhall, Ph.D., Associate Professor of Sociology and African 
American Studies, University of Illinois—Urbana—Champaign 


Dr. Mendenhall described the Racial Microageression Research Project, which defines racial 
microageressions as brief, commonplace, daily, verbal, behavioral, or environmental 
indignities—intentional or unintentional—toward people of color. Her research aimed to 
explore the impact of racial microageressions on students at a predominantly white 
university. When the study began in 2007, there were few measures of the prevalence of 
racial microageressions. Dr. Mendenhall and the other researchers developed an online 
sutvey that was informed by qualitative data and also captured additional qualitative findings. 


Dr. Mendenhall said the online survey provided some surprising information, including the 
first descriptions she and her colleagues had directly from students about their experiences. 
Focus groups provided additional details. The findings were compiled into a report that 
received broad media coverage. As a result, student voices were heard, and the community 
learned about the impact of racial microageressions. 


From both the racial microaggressions research and her other qualitative research, Dr. 
Mendenhall learned that organizing qualitative data for analysis (e.g., transcribing and coding 
interviews) is time-consuming, challenging, and expensive. However, qualitative data provide 
insight into hidden processes and individuals’ decision-making strategies. 


Mario Luis Small, Ph.D., Professor of Sociology, Harvard University 


Dr. Small and colleagues studied social networks among mothers whose children were 
enrolled in child care centers in New York City. The large study encompassed four 
neighborhoods, representing a range of demographics and economic classes, and used 
sutveys, interviews, and observations to gather information. It also included quantitative data 
from the national Fragile Families and Childhood Wellbeing Study (see 

http://www. fragilefamilies.princeton.edu). 


Through his work, Dr. Small found that variations in the level of networking among 
mothers depended on the individual child care center, not the neighborhood. The strongest 
social networks emerged at child care centers that had the strictest rules around picking up 
and dropping off children (such as significant monetary fines for lateness), the most 
numerous field trips and similar activities requiring parental participation, and otherwise the 
most opportunities or inducements for parents to interact. In such settings, all of the parents 
are at the child care center at the same time, and they begin to turn to each other for help in 
avoiding fines when work or other barriers prevent them from picking up or dropping off 
their children on time. Analysis of survey data about well-being, depression, and other 
factors led Dr. Small to conclude that those who developed a social network through their 
child care center were better off than those who did not. 
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Dr. Small contends that without the qualitative research, he may not have seen that 
something small, like pick-up and drop-off policies, could have a big effect. Dr. Small said 
inductive work—seeking to learn what is not already known—is an important part of 
science. Different kinds of data produce different kinds of evidence, so MMR offers a 
complementary approach in which both qualitative and quantitative data are equally valuable. 


Mixed Methods Measurement Research (Goal 5) 
Sandra Chafouleas and Tom Weisner 


Sandra Chafouleas, Ph.D., Associate Dean for Research, Professor, 
Department of Educational Psychology, University of Connecticut 


Dr. Chafouleas described the development of the Direct Behavior Rating (DBR) scale as a 
mechanism to gather real-time data about student behavior, such as to monitor the progress 
of an intervention in a classroom. The project uses MMR approaches, such as a verbal 
protocol analysis (or think-aloud), single-case analysis of teacher problem-solving behavior, 
and teacher surveys around usability, to better understand how teachers use the DBR scale. 
Although many psychometric studies have been conducted to support DBR use in screening 
and progress monitoring, psychometric data alone cannot provide all relevant information, 
such as revealing detail how teachers use a tool to make decisions while in the classroom. 
Echoing Dr. Small, Dr. Chafouleas said that qualitative and quantitative findings are 
complementary. 


Achieving the goals of the project requires a strong team with broad expertise and the ability 
to communicate. Dr. Chafouleas said it can be difficult for investigators from different fields 
to find common language at times. Also, mixing methods of research and using a team 
approach to do so takes a lot of time, yet it is critical to a good project. 


Dr. Chafouleas said there should be guidance from IES and in requests for applications 
(RFAs) on how MMR works. However, investigators also should be encouraged to build 
strong teams that can provide all the needed expertise to ensure a successful project. 


Tom Weisner, Ph.D., Professor of Anthropology, University of California— 
Los Angeles 


Dr. Weisner emphasized that the world is not linear, additive, or decontextualized; and at the 
same time, there is no question that the world can be usefully modeled as if it were. 
Therefore, the question for researchers is not whether to use MMR but rather why not use 
it. In his view, as an aspirational research methods framework at least, MMR should be the 
default plan — the unmarked option. Researchers would need to provide a rationale for why 
they had not included MMR, where feasible, in their proposals. As previous speakers noted, 
using multiple methods can provide new information or enhance understanding. The goal as 
well as the ultimate rationale of MMR should be to gather information that could not have 
been revealed without such an approach. 
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Research proposals should incorporate enough flexibility for investigators to examine 
unexpected findings and unintended consequences. Moreover, someone from the research 
team must be in the field, engaged in the context of the research, to recognize events or 
factors that might influence the research. Dr. Weisner described how the qualitative research 
(e.g., interviews in the home) in his study of interventions to help the working poor revealed 
why some families did not take up the interventions offered. These data were also essential, 
in conversation with quantitative data, to account for many findings and to suggest new 
analyses and fieldwork questions. 


In another study, by talking in depth with families of children with disabilities, Dr. Weisner 
and colleagues found that measuring the level of “hassle” families faced on a daily basis and 
their perceptions of sustainability of their family’s routines were as good at predicting family 
satisfaction as more complicated assessment tests, such as parental reports of coping or 
stress. The same study identified new ways parents could help their children with disabilities 
cope and communicate about their disabilities. 


Dr. Weisner observed that findings that matter come from a combination of methods. In 
effective research teams, each member, regardless of his/her methodological expertise, 
should have some understanding of the topic of the research so that members can 
communicate effectively with each other and also with the subjects in the community. Using 
MMR 1s labor-intensive, and investigators using qualitative methods in context deliberately 
cede some control to the subjects, said Dr. Weisner. The discovery of findings that would 
not have been obtained otherwise then justifies this openness through researchers “being 
there” in the community. Such research can be difficult to publish in some journals and 
books. However, having a pluralistic publication strategy targeting places that do publish 
MMR with others, is effective. Dr. Weisner believes that every step of research—including 
the theory/conceptualization, design, sampling, analysis, and even implementation—should 
take advantage of MMR, both quantitative and qualitative. 


Mixed Methods in Efficacy/Effectiveness Research (Goals 3 and 
4) 
Carolyn Heinrich and Patricia Burch 


Carolyn Heinrich, Ph.D., Sid Richardson Professor of Public Affairs, 
Director, Center for Health and Social Policy, University of Texas—Austin 


Dr. Heinrich said that although she had no formal training in qualitative research methods, 
she found that interviews, focus groups, and discussions enriched her empirical, quantitative 
research. Dr. Heinrich and Patricia Burch, Ph.D., coauthored a book advocating the 
integration of MMR throughout research design, designation of objectives, measure creation, 
model selection, execution, interpretation, and analysis. 


Dr. Heinrich noted that incorporating mixed methods in the field as opportunities arise led 
to findings being better received and more credible—increasing their uptake. Thus, MMR 
improves the dissemination of research, she said. Successful MMR requires strategies for 
team-building and resource management. 
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From her own work, Dr. Heinrich saw that selection takes place even in purportedly 
randomized approaches, and qualitative research is important for understanding selection. 
For example, a national study of tutoring used a measure applied by many schools: invoiced 
hours of instruction. But a closer look revealed that schools in different cities had 
substantially different ways of calculating the measure. Similarly, research will determine an 
average impact, but investigators should look at the range of impacts and understand how 
those ranges correlate with various factors. Like Dr. Weisner, Dr. Heinrich emphasized the 
need for flexibility in research design because the factors affecting the impact of an 
intervention will not always be apparent in the design phase. 


At present, materials describing MMR tend to lean heavily toward one type of research and 
are not reflective of the field, said Dr. Heinrich. Students and early-career investigators need 
a better understanding of how research evolves and how to plan to leverage the advantages 
of MMR throughout their research. 


Patricia Burch, Ph.D., Associate Professor of Education, University of 
Southern California 


From the evaluation of the effectiveness of tutoring programs, Dr. Burch offered examples 
of weaving good MMR into every stage of research, specifically the integration of ELLs into 
tutoring. Qualitative research can identify the conditions that support or stimulate 
participation in tutoring programs, for example. 


In the course of research, it became clear that online, for-profit tutors were gaining market 
share. Dr. Burch and colleagues aimed to create a model of online tutoring to see the effects 
but found significant variation in what was considered “online.” Dr. Heinrich added that the 
discrepancies in definitions and classifications of online tutoring could change thinking 
about the topic. 


Dr. Burch agreed with previous speakers that MMR is an iterative process that requires 
teamwork. She called for a discussion of ways to encourage collaboration among people with 
shared expertise on how to get up to speed on applying MMR—for example, through IES 
summer research training institutes. 


Dr. Burch said her research endeavors embedded investigators in each school district and 
invested in dissemination strategies for communicating findings to policymakers and others. 
She suggested building such approaches into guidelines, training, and RFAs. 


Finally, Dr. Burch said that training programs often focus on either qualitative or 
quantitative research. Ideally, an introductory course to MMR could focus on tasks that 
bridge the two approaches, such as verifying and assessing data. Dr. Heinrich added that an 
IES focus on bridging quantitative and qualitative analysis could break down boundaries. 
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Mixed Methods in Development Research (Goal 2) 
Jill Hamm and Jennifer Russell 


Jill Hamm, Ph.D., Director, Social Development Intervention Research 
Program, Associate Professor of Educational Psychology, Measurement, 
and Evaluation, University of North Carolina—Chapel Hill 


Dr. Hamm described the launch of a successful intervention delivered to sixth-grade 
teachers to help adolescents adjust to middle school. She and her colleagues are now seeking 
to revise the intervention for seventh- and eighth-grade teachers. It includes site visits, 
observations, and interviews; a summer institute for teachers; and content-based modules. 
The revised intervention includes a “scouting report,” based on classroom observations and 
interviews, to generate feedback to investigators, better identify the points of leverage in the 
classroom, and guide the intervention. 


Qualitative research approaches identified challenges in implementing the intervention and 
pieces that should be added or deleted, said Dr. Hamm. Some methods were not part of the 
original research proposal. For example, because investigators forged close partnerships with 
schools to test the sixth-grade intervention, they now have data they can use for comparison 
when they apply the intervention to seventh-graders. Those data can shape the content and 
delivery of the intervention along with the scouting report, Dr. Hamm noted. By collecting 
the same data for the seventh-grade intervention, researchers hope to track changes among 
students, particularly with respect to behaviors and contextual factors targeted for attention 
from data collected from students during the middle school transition year. 


Using MMR helped the research team identify emerging intervention needs and identify 
unexpected and valuable factors—and to understand them in context, said Dr. Hamm, 
which improved delivery of the intervention. Qualitative and quantitative data 
complemented each other and enriched understanding. However, it can be difficult to 
analyze quantitative data quickly (e.g., over the summer) so that it can be applied 
meaningfully to an intervention. Researchers face challenges publishing the findings of 
MMR, Dr. Hamm concluded. 


Jennifer Russell, Ph.D., Associate Professor and Research Scientist, 
University of Pittsburgh 


Dr. Russell described a researcher—practitioner partnership with the goal of refining a model 
for mathematics coaching to transition to Common-Core-aligned instruction, ultimately 
scaling up the approach for deployment across the state of Tennessee. In this project, the 
team gathers a lot of data (quantitative and qualitative) that it uses to feed rapid continuous 
improvement cycles. 


The initial model emphasized deep and specific coach-teacher conversations about the 
instructional triangle: the nature of the math being taught, what student thinking looks like, 
and teachers’ pedagogical choices. However, Dr. Russell noted, feedback emerged from 
qualitative analyses as another important aspect of coaching—specifically, the characteristics 
of the feedback that coaches provided to teachers. The research team is now considering 
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how to incorporate that feedback into its quantitative models and measures and how to 
work it into guidelines to support effective coaching and feedback. 


The team is also using MMR to see how its model works in different contexts. Dr. Russell is 
doing similar research using collaboration and rapid cycles of improvement to assess how 
schools support students with special needs. She said collaborative design work is fertile 
ground for MMR because it helps investigators understand the nature of the problem and 
the role of context in shaping implementation and understanding outcomes. Dr. Russell 
emphasized that teams should include practitioners—both in the field and in the 
policymaking realm—who can speak to issues of feasibility, the role of the school district, 
and the role of the state. 


Dr. Russell said her team faces challenges in analyzing data for rapid cycles of improvement. 
Some of the analysis using well-established measures can be done by expert 
raters/consultants, but exploratory qualitative analyses that contribute to development of 
new measures must be addressed internally by the research team. 


Opportunities and Challenges in Training Researchers to Use 
and Integrate Mixed Methods 
Meredith Larson, Research Analyst, NCER 


Following the discussion of how MMR fits into the IES goal structure, the meeting turned to 
the topic of researcher training. Dr. Larson briefly outlined the predoctoral and 
postdoctoral training programs currently supported by NCER and NCSER. She also noted 
the methods training programs (contracts and grants) that have evolved in recent years to 
address specific research designs, such as randomized controlled trials, single-case design, 
and cost analysis; and NCSER’s Early Career Development and Mentoring program. 


Dr. Larson estimated that NCER and NCSER have trained about 800 predoctoral and 
postdoctoral fellows, plus a few hundred more people through workshops and summer 
programs. She did not know of any postdoctoral programs focused primarily on qualitative 
methods or MMR training. 


Pre- and Post-Doctoral Training in Mixed Methods 
Mike Knapp and Sandy Chafouleas 


Mike Knapp, Ph.D., Professor, Educational Leadership and Policy Studies, 
Director, Center for the Study of Teaching and Policy, College of Education, 
University of Washington—Seattle 


Dr. Knapp described an interdisciplinary, pre-doctoral training program at his institution, 
which focused on MMR in education research and its application to policy issues across the 
P-20 spectrum. The faculty included scholars actively working in all types of research 
methodology and representing the disciplines of education, public affairs, sociology, and 
economics. The program trained 24 doctoral students from the Education, Sociology, 
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Economics, and Public Policy departments. Dr. Knapp stressed that, to help students gain 
confidence in research, the program had to cover sophisticated quantitative research, 
rigorous qualitative research, and integrated, complementary MMR, through a set curriculum 
that addressed all three, alongside substantive coursework that met home department 
requirements and enabled them to pursue their respective specializations. 


Throughout their 4 years of study, the students had regular colloquia with scholars of 
different perspectives and also met in an integrated pro-seminar every 2 weeks. These 
opportunities offered a lot of time to wrestle with issues. Students were required to attend 
three conferences a year (with fellowship travel support) and to present at conferences after 
their first year. Dr. Knapp said that given resources and encouragement, the students rose to 
the challenge, and some even published their early research. 


Throughout the program, students took part in research apprenticeships with faculty (their 
own advisors or others), though not all of the students received rich experiences in both 
qualitative and quantitative research in their apprenticeships. Finally, students were 
encouraged to use MMR in their dissertations, which is especially challenging for students 
with no research team or limited funding. However, about half succeeded in producing a 
dissertation reflecting a good mix of research types. Dr. Knapp said students saw themselves 
as MMR investigators and looked for research questions that invited MMR approaches. 


The program spun off a graduate certificate program in Mixed Methods Education Sciences, 
available to any Ph.D. students in the university meeting the program requirements. It relies 
on the core coursework of the pre-doctoral program plus a capstone project. In addition, the 
certificate program has seen good results from its year-long course in rigorous qualitative 
reseatch that culminates in preparation of a proposal for a conference. Many students have 
succeeded in presenting at the American Educational Research Association conference, and 
some have published their research. Dr. Knapp said immersion is key; the students with 
experience in quantitative research were the most resistant to learning qualitative methods 
but also have been the ones most engaged by the end of the course. 


Dr. Knapp summarized the lessons learned: 

e Students need training in all three approaches: quantitative research, qualitative 
research, and MMR. Even if they ultimately focus on one approach, learning all three 
is key to developing a common language among research teams. 

e Careful mentoring is key to spotting, nurturing, and growing good MMR 
investigators. Students need hands-on practice in MMR. 

e Learning takes a community where students can have in-depth conversations with 
peers and faculty, and building those relationships can take years. An interdisciplinary 
team of interested faculty is essential to keeping the conversation going. 


Dr. Knapp suggested paying attention to what it means to be “bilingual’’—that is, fluent in 
both qualitative and quantitative approaches—and how that definition changes depending 
on the nature of research teams and partnerships. The answers to these questions could 
affect how training programs are created. 
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Sandra Chafouleas, Ph.D., Associate Dean for Research, Professor, 
Department of Educational Psychology, University of Connecticut 


Dr. Chafouleas said her training program had four postdoctoral students and was designed 
to increase the quality of special education research science, with a strong focus on research 
methods. The students had individualized fellowship plans. Faculty in the Center for 
Behavioral Education and Research worked with them to update the plans every semester to 
address needs, identify research opportunities, and develop a research agenda. The MMR 
approach was accidental rather than purposeful, said Dr. Chafouleas. 


Individual planning is critical, Dr. Chafouleas stated. Postdoctoral students have more 
flexibility to think about where their work is going and to determine what they need to 
support their goals. Unlike predoctoral students, they can seek varied opportunities (e.g. take 
as many ot few classes) under this fellowship program. Students were required to seek IES 
summer training programs as well as other training, with support from the university. Dr. 
Chafouleas said that working with students outside of a prescribed pathway is essential to 
training. 


As Dr. Knapp said, bringing people together who are willing to talk is important, said Dr. 
Chafouleas, and faculty must facilitate such discussions. Training grants are a good way to 
stimulate conversation, she noted. Dr. Knapp added that in his program, faculty came from 
different perspectives and had to find a way to talk about MMR. In good research teams, he 
noted, the investigators learn to talk across their differences and understand each other. 


Discussion of the Future of Mixed Methods in Education 
Research 


Both during and following the presentations, participants discussed the challenges and 
opportunities of MMR. Participants’ observations and suggestions, summarized below, 
include general observations for MMR in education research, potential IES activities to 
promote MMR (which IES could incorporate into guidance or training), and suggestions for 
researchers and the field. Currently, the bullets under each subheading are in no particular 
ordet. 


General Observations about Mixed Methods Research 

e There is consensus that qualitative data is valuable, enriches quantitative data, and 
provides insight that cannot be gained from quantitative research alone. 

e Qualitative and quantitative data are complementary, and both can inform the 
iterative process of research. 

e Qualitative research can help identify possible causal relationships, especially when 
coupled with quantitative data in a strong design that allows causal inference. It can 
also reveal the underlying mechanisms or processes that lead to program effects. 

e Researchers need to look at a lot of data from different perspectives to uncover and 
understand causal factors. 

e Good field work can provide insight even when randomized controlled trials suggest 
an intervention is not effective. For example, qualitative research can shed light on 
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how students are responding to instruction or other services offered to them and on 
their level of engagement. It can also shed light on the context in which services are 
delivered, including what services control group members may be receiving. 
Investigation into the “black box” of the treatment and into services offered to 
controls provides essential information on how the program worked or why it did 
not for some and did for others. 

Looking at the average impact only can mask the range of impacts. In special 
education research, average impacts may not even be relevant, given small n’s and 
the nature of the risk/disability. The goal for special education is often to 
individualize services to meet student needs. 

Good qualitative research requires a lot of relationship-building and trust. 

To optimize the design of a mixed methods study, applicants should plan carefully to 
include quantitative and qualitative expertise on the research team from the very 
beginning of the study. 

Incorporating MMR into the research design offers a way to build in flexibility to 
anticipate unexpected findings, investigate failed hypotheses, or dig deeper into 
inconclusive findings. MMR may also prove useful to investigating unintended 
outcomes of a program or policy. 

Specifying a hypothesis, formulating research questions, and analyzing data are 
interrelated; focusing too much on a single method contributes to a narrow, limited 
interpretation of the data. 

The steps required to formalize qualitative research (e.g., institutional review board 
approval and documentation of individual consent) can pose a barrier. 

Much of the work that goes into developing a study—e.g., informal conversations, 
understanding of the community, feedback, and refinements to processes—is 
inherently qualitative and should be considered part of the formal data collection, as 
it is often useful in providing context for the results. 

Review the qualitative data first (interviews, transcripts, field observations) by simply 
indexing or marking their content to review for overall patterns, then select a subset 
of materials for deeper quantitative analysis (e.g., formal coding with reliability). 
Gathering and integrating qualitative and quantitative data are expensive and time- 
consuming processes. 

There are already journals with a focus on MMR, but the topic of education research 
using MMR is a ripe for special issue in one of these journals—in print, online, or 


both. 


SUGGESTIONS FOR IES 


RESEARCH 


Signal to applicants and reviewers that high quality MMR is encouraged in proposals. 
IES should use the term “MMR” explicitly. In RFAs, applicants will make the case 
for what constitutes MMR. It should be noted that MMR is not synonymous with 
qualitative research. 

Craft the language of RFAs to encourage potential applicants to think about 
alternative ways to answer research questions. 


Mixed Methods in Education Research IES TWG | Meeting Summary 13 


Avoid being too prescriptive, as there is little agreement in the field about the 
optimal methods. 

Provide some exemplars to demonstrate different approaches to MMR. These could 
come from IES-funded proposals or from existing guidelines from NIH or others. 
Provide real-world examples of challenges to MMR and how they were managed. 
Create webinats, videos, or documents in which individual researchers describe their 
experiences—e.g., their moments of discovery and next steps. 

Produce videos or other guidance on how to incorporate MMR into proposals. 
Encourage applicants to include examples of the kinds of measures that may be 
used, demonstrating that they have thought carefully about protocols and options for 
data collection. 

Suggest that applicants describe how they would handle unexpected findings, 
perhaps drawing on other studies to indicate what they might do so (similar to the 
way RFAs require applicants to discuss how they would address missing data). 

Give guidance on how to present the methodology in an organized manner that 
reviewers can follow (particularly for proposals addressing Goal 2). 

Recommend that applicants show the value of MMR throughout the proposal, not 
just in the methods section. For example, does the literature review draw upon both 
qualitative and quantitative research; do the research questions lend themselves to 
different methodologies; does the research team include individuals with diverse 
methodological backgrounds and training? 

Include in RFAs examples of funded projects using MMR that reflect the content 
IES wants to see. 

Direct potential applicants to abstracts (online) that describe MMR methods and 
designs that were funded. 

Develop guidelines around the principles of MMR that can be used to help 
researchers determine how to proceed. 

The language of Goal 1 currently seems to favor quantitative analysis. It could be 
changed to included language that encourages MMR. 

Because MMR typically takes a long time, consider incentives that will encourage 
researchers to invest in MMR for the long term. For example, encourage researchers 
to build on their existing body of work instead of always seeking to create new 
collaborations for each new grant opportunity (e.g., continue promising projects, 
rather than needing to come up with new projects and teams for new grant 
applications). 

Raise awareness about additional funding or other mechanisms available for further 
analysis of findings. 

Clarify for reviewers that proposals can and should describe some flexibility in 
research design so that investigators can respond to issues that arise. Proposal 
reviewers should consider whether the investigator anticipated challenges or barriers 
and how to address them. 

Create more incentives for collaboration among investigators who use different 
research methods. 
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Use the annual gathering of IES-funded investigators to facilitate discussion across 
methodological disciplines, e.g., a workshop on lessons learned or opportunities for 
researchers to seek input from peers. 

Consider gathering researchers who are using MMR in some way so that they are 
aware of and can learn about measures, sampling, analyses, etc. from one another. 


TRAINING 


Build the pipeline of future researchers by providing MMR training, e.g., through a 
summer institute or longer training program. 

To grow the cohort of investigators with experience in MMR, provide funding for 
assistant professors to take part in data reanalysis efforts using MMR, so they can 
learn and carry that knowledge forward. 

Develop an RFA dedicated to training in MMR. 

A 1-week workshop provides sufficient time for students and early-career 
investigators to meet experienced researchers, discuss real-world examples, and 
address research planning. It is also an appropriate length of time for a researcher to 
develop the communication skills to join a large MMR team (but not enough time to 
gain expertise in MMR). 

Identify the institutions that have established a reputation for MMR in education 
science and encourage them to develop networks and collaborations. 

Provide funding for researchers to train at institutions with established MMR 
programs. 

Provide funding for faculty to work with established MMR training programs on 
setting up their own MMR training programs at their home institutions. 

Adopt a similar model to that of the NIH to provide supplemental funding to 
existing projects, in this case providing resources for training for extending research 
and leveraging investments already made in projects using MMR. The NIH model 
for example, is the minority supplement grant program, but in this adaptation for 
MMR//IES, funded investigators could apply for additional funds for postdocs or 
graduate students who can extend qualitative or quantitative methods in their 
research to encourage MMR analyses or supplemental data collection. 

Create short training videos for students with researchers talking about their 
experiences in actual MMR projects. Explain, for example, how to anticipate and 
adjust research methods in response to unanticipated challenges. 

Create focused educational modules similar to those developed by SAGE 
Publications (see https://stmo.sagepub.com/browse?doctype=qass). Partner with 
SAGE to develop a short series on MMR research methodology (e.g., as part of its 
Little Green Books series on research methods). 

Look at materials already being used for training elsewhere and consider how to 
publish or disseminate them mote broadly. 
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SUGGESTIONS FOR RESEARCHERS AND THE FIELD 


e In research applications, account for the time needed to collect and interpret 
qualitative data and to gather and clean quantitative data when presenting your 
research design. 

e Involve communities, school districts, and practitioners in the design process so that 
they understand what the research involves and can provide input on what is 
meaningful and feasible. A partnership research model includes all these stakeholders 
from the time of the conceptualization of a project through to publication and 
dissemination and implementation. 

e Maximize the use of data analytics (i.e., the examination of raw data to make 
conclusions about the data) to get real-time data, for example, which could preserve 
rigor and mitigate the costs of qualitative research. 

e Invest in resources to improve the technology (e.g., better software for transcribing 
interviews). Consider attending workshops on current MMR software use. 

e Encourage journals that emphasize MMR, such as Journal of Mixed Methods Research. 
International Journal of Multiple Research Approaches, Quality and Quantity, and Field 
Methods, to publish a special issue on education. Investigators can propose that 
journals include the methodological details and data in an appendix, which can be 
posted online. 

e When qualitative data identifies possible causal factors of quantitative findings, the 
published MMR findings should describe the process and results in one document 
that both qualitative and quantitative researchers can read. 

e The detailed results of MMR can be written up as a series of articles and published 
sequentially (see above). 

e Break down the silos between quantitative and qualitative research. Expose students 
to both types of research, so they can begin thinking about bridging the two. There 
are often disciplinary as well as methodological silos. The purpose of methods 
training should be on preparing people to use the most appropriate methods for 
particular research questions, not just on the method itself. Decontextualized training 
purely focused on a method, regardless of what method, is less beneficial. 

e Students typically are not trained in both qualitative and quantitative research 
methods in their doctoral programs. Even when they are exposed to both, they rarely 
receive training on integrating the two approaches. 

e Encourage students to look at MMR as a complementary or secondary area of 
expertise. Recognize that primary expertise in a particular methods skill set is 
important at the time of graduate training and in early careers, but the ability to use 
MMR in teams and as supplemental methods work is also of career-long value. Such 
MMR skills can lead to funding. 

e Steer students toward research apprenticeships that use MMR in the field. 
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Wrap Up and Next Steps 
Thomas Brock, Ph.D., Commissioner, NCER, and Joan McLaughlin, Ph.D., 
Commissioner, NCSER 


Dr. Brock was encouraged by participants’ articulation of the value of MMR and its relevance 
to all five IES research goals as well as the researcher—practitioner partnership programs. He 
was struck by the many examples of how MMR may be integrated into every phase of 
reseatch. He also was struck by the observation that different emphases may be given to 
qualitative and quantitative methods in the course of a study, but what is important is that 
there be regular dialogue between researchers who specialize in the two methods. Finally, he 
appreciated the practical suggestions offered to IES, and noted several that may be 
implemented in the short term, such as: 


e Make examples of good MMR funded by IES readily available. 

e Raise awareness about examples of good MMR and keep the conversation going in 
the field (e.g., through social media). 

e Host a workshop on MMR during the IES investigators’ meeting in December. 

e Communicate with investigators who have training grants about steps they can 
incorporate into their classes now. 

e Incorporate questions into the next round of RFAs that encourage a thoughtful 
approach to MMR. 

e If funding is available, create a new grant opportunity around MMR. 


Dr. Brock welcomed further suggestions and encouraged participants to contact IES staff at 
any time. Dr. McLaughlin also thanked the participants for their input. The meeting 
adjourned at 3:33 p.m. 
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Appendix A 


UNITED STATES DEPARTMENT OF EDUCATION 


INSTITUTE OF EDUCATION SCIENCES 


National Center for Education Research 
National Center for Special Education Research 


Meeting Objectives for May 29, 2015 
Dear Colleague, 


Thank you for participating in the Mixed Methods in Education Research Technical Working Group 
([WG). This TWG meeting is being hosted by the National Center for Education Research (NCER) 
and the National Center for Special Education Research (NCSER).? These two research centers 
within the Institute of Education Sciences (IES) fund a wide range of education research projects, 
including some which use “mixed methods” (defined for the purposes of this meeting as combining 
qualitative and quantitative research methods). To date, however, IES has not provided much 
guidance on mixed methods in its grant application materials or on its website. Not surprisingly, 
some reseatchers have misconceptions or confusion about the kind of research that IES funds. Some 
believe that IES does not fund any qualitative research. Others who propose to include qualitative 
methods in their applications to IES grant competitions. Finally, some IES reviewers are uncertain 
about how to evaluate proposals that include qualitative struggle with how much or what kind of 
information to include methods. 


In the types of education research that evaluate program effects (e.g., randomized controlled trials, 
single case designs), IES has been a leader in establishing methodological standards and providing 
resources for researchers to use when they design and conduct studies. For example, NCER, 
NCSER, and the National Center for Education Evaluation (NCEE, also within IES) publish 
methodological papers that are free for download, and the What Works Clearinghouse (WWC, 
funded by NCEE) regularly conducts free webinars on evaluation design. NCER and NCSER also 
support summer workshops and pre- and post-doctoral training programs that focus on research 
methods. 


TES is seeking input from the TWG about whether IES can or should play a similar role in qualitative 
research methods in education. Possible options may include developing standards or offering 
guidance to the field on issues such as when qualitative methods might be considered, how to ensure 
the data gathered through qualitative methods are accurate and reliable, and/or how to integrate the 
findings from qualitative research into quantitative studies. In addition, because IES funds education 
research training, it would be helpful to understand whether there is a need for training on the use of 
mixed methods for education research and, if so, what kinds of training are needed and for whom. 
We are open to any and all suggestions, such as the following: 


a) New publications on the use of mixed methods; 
b) New methodological tools; 


? National Center for Education Research (NCER, http://ies.ed.gov/ncer) 
National Center for Special Education Research (NCSER, http://ies.ed.gov/ncser) 
3 National Center for Education Evaluation and Regional Assistance (NCEE, http://ies.ed.gov/ncee) 
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c) New or clearer instructions for applications to our grant programs; 
d) New funding opportunities; 

e) New training opportunities; and/or 

f) Other ideas we have not considered. 


We welcome your ideas and feedback, and look forward to a lively discussion. 
Sincerely, 


Thomas Brock, Commissioner, NCER 
Joan McLaughlin, Commissioner, NCSER 
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Appendix B 


OVERVIEW OF RESEARCH AT THE INSTITUTE OF EDUCATION 


SCIENCES 


The Institute of Education Sciences (IES) is the research arm of the U.S. Department of 
Education. Established by the Education Sciences Reform Act of 2002, IES’s mission is “to 
provide national leadership in expanding fundamental knowledge and understanding of 
education from early childhood through postsecondary study.” IES carries out its activities 
through four Centers: 


The National Center for Education Statistics (NCES) is the primary federal 
entity for collecting and analyzing data related to education in the U.S. It fulfills a 
Congressional mandate to collect, analyze, and report complete statistics on the 
condition of American education; distribute information and publish reports; and 


review and report on education activities internationally. 


The National Center for Education Evaluation and Regional Assistance 
(NCEE) conducts unbiased large-scale evaluations of education programs and 
practices supported by federal funds; provides research-based technical assistance to 
educators and policymakers; and supports the synthesis and wide spread 
dissemination of the results of research and evaluation throughout the United States. 
NCEBE?’s responsibilities include management of the Regional Education 
Laboratories, the What Works Clearinghouse, and the Education Resources 
Information Center (ERIC). 


The National Center for Education Research (NCER) sponsors research that 
addresses the nation’s most pressing education needs, from early childhood to 
postsecondary and adult education. 


The National Center for Special Education Research (NCSER) sponsors a 
rigorous and comprehensive program of special education research designed to 
expand the knowledge and understanding of infants, toddlers, and students with or 
at risk for disabilities from birth through high school. 


The Technical Working Group meeting will focus exclusively on the work of the two 
Research Centers: NCER and NCSER. As background, this document provides an 
overview of the research and training programs that are offered by these Research 
Centers along with some details on the number and amount of awards made by NCER 
and NCSER since they have been in operation. 
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Research and Training Programs Offered by the Centers 


The Research Centers accomplish their missions primarily by making grants to education 
researchers. Anyone can apply for a grant, but all proposals must go through a rigorous 
peer-review process, and only the projects that reviewers have rated outstanding or excellent 
will be considered for funding. In the past, the Institute was able to fund all projects that 
received outstanding or excellent ratings (9 to 13 percent of the proposals reviewed in FY 
2011 and FY 2012). Since the budget sequestration in FY 2013, the Institute has not had 
sufficient funds to award grants to all the projects that received outstanding or excellent 
ratings, and an even smaller percentage of reviewed applications have recetved awards. 


The largest and best-known grant programs operated by the Centers are the Education 
Research Grants run by NCER, and the Special Education Research Grants run by 
NCSER. The goals of both programs are to support research that will build knowledge 
leading to improvements in the quality of education policies, programs, and practices and 
ultimately in student outcomes. NCER’s Education Research Grants program focuses on 
typically developing students from age three through adult and postsecondary education; 
NCSER’s Special Education Research Grants program targets infants, toddlers, children, and 
youth with or at risk of disabilities. 


To apply for an Education Research Grant or a Special Education Research Grant, 
researchers must identify both a research topic and a research goal. Each year, NCER and 
NCSER provide a list of research topics that are meant to focus grant applications to the 
areas of most critical need for research (a complete list for each center is included at the end 
of this document). Within these broad topic areas, grant applicants may propose to address 
research questions that they identify as the most significant for the field. Under all topics, all 
researchers must address student outcomes (e.g., academic achievement, acquisition of social 
ot behavioral skills, and high school completion). 


The Education Research Grants program and the Special Education Research Grants 
program currently support five research goals: 


1) Exploration. Research supported under the Exploration goal identifies (a) 
malleable factors that can be changed by the education system and are associated 
with education outcomes for students, such as children’s behaviors or teachers’ 
practices; and (b) factors and conditions that may mediate or moderate the relations 
between malleable factors and student outcomes. This identification is to be done 
through the analysis of data (collected by the project and/or using a secondary data 
set) or the meta-analysis of research studies. 


2) Development and Innovation. Research supported under this goal develops 
innovative education interventions and improves existing education interventions 
that are to produce beneficial impacts on student outcomes when implemented in 
authentic education delivery settings (e.g., classrooms, schools, districts). IES 
considers interventions to encompass curricula, instructional approaches, 
technology, education practices, programs, and policies. An iterative process is used 
in the development of the intervention and a pilot study is done to determine if there 
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is evidence of the promise of the intervention for achieving its intended student 
outcomes. 


3) Efficacy and Replication. Research supported under the Efficacy and 
Replication goal determines whether or not fully developed interventions produce a 
beneficial impact on student outcomes (and the practical importance of that impact) 
relative to a counterfactual when implemented in authentic education delivery 
settings. Interventions can be implemented under ideal conditions, which may 
include use of greater implementation support or a more homogeneous sample than 
would be expected under routine practice. Efficacy and Replication projects are to 
provide causal analysis, and randomized controlled trials are the favored research 
design, though strong quasi-experimental designs can be used. 


4) Effectiveness. Research supported under the Effectiveness goal determines 
whether or not fully developed interventions with prior evidence of efficacy produce 
a beneficial impact on education outcomes for students relative to a counterfactual 
when they are implemented under routine practice in authentic education delivery 
settings. Effectiveness projects are to provide a causal evaluation of an intervention 
as well as examine the intervention’s fidelity of implementation. In addition, the 
evaluation team must be independent from the developer/distributor of the 
intervention. 


5) Measurement. Research supported under the Measurement goal supports (a) the 
development of new assessments or refinement of existing assessments and the 
validation of these assessments or (b) the validation of existing assessments for 
specific purposes, contexts, and populations. 


Depending on the research goal, applicants may request funding for 2 to 5 years. Individual 
grants typically range from $200,000 for exploration projects using secondary data to $5 
million for large-scale effectiveness trials. 


In addition to the Education Research Grant and Special Education Research Grant 
Programs, NCER and NCSER occasionally run competitions for National Research and 
Development (R&D) Centers. The purpose of the R&D Centers is to contribute 
significantly to the solution of education problems in the United States by supporting a 
coordinated set of research, development, evaluation, and national leadership activities aimed 
at improving the education system and, ultimately, student achievement. The grants are 
typically larger than in the other competitions (about $10 million) and extend over 5 years. 
NCER and NCSER often use the R&D Centers as a vehicle to focus research attention on 
new of emerging issues or particularly challenging problems in education. NCER, for 
example, recently established an R&D Center to test the effectiveness of innovative 
assessment and instructional practices designed to help the large number of entering college 
students who are placed into developmental (or remedial) classes. Similarly, NCSER 
recently funded an R&D Center to a research team that is developing and evaluating the 
efficacy of a comprehensive school- and community-based treatment model for high school 
students with Autism Spectrum Disorder. 
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A continuing concern at IES is to make sure that the research it funds is relevant and useful 
to policymakers and practitioners. In support of this goal, the Institute recently developed a 
grant program called Partnerships and Collaborations Focused on Problems of 
Practice and Policy. The fundamental idea behind this program is to support projects in 
which policymakers and practitioners have defined the research question of interest and to 
encourage researchers and practitioners to work together to develop research plans, collect 
data, and disseminate results. The program offers three types of grants: (1) Researcher- 
Practitioner Partnerships in Education Research, intended to support newly formed 
partnerships to begin conducting research and develop future research plans; (2) Continuous 
Improvement Research in Education (CIRE), designed to support efforts by relatively 
mature partnerships to implement, adapt and revise an approach to addressing an education 
issue or problem; and (3) Evaluation of State and Local Education Policies and Programs, 
designed to support projects examining the impact of fully developed policies and programs 
implemented under routine conditions in authentic education settings. The amount and 
length of funding ranges from $400,000 over 2 years for a Researcher-Practitioner 
Partnership grant, to $5 million over 5 years for an Evaluation of State and Local Education 
Policies and Programs grant. NCER manages the program, but applicants may propose 
projects that are focused on typically developing students or on students with or at risk of 
disabilities. 


NCER and NCSER operate a Small Business Innovation Research (SBIR) Program 
that provides up to $1,050,000 in funding to small business firms and partners for the 
reseatch and development (R&D) of commercially viable education technology products. 
The products must be designed to improve teacher and/or student outcomes in the fields of 
education or special education. 


NCER tuns a competition for Statistical and Research Methodology in Education. 
This program supports the development of tools and procedures that will enhance the 
capacity of researchers to design high-quality studies and conduct statistical analyses. 
Applicants may request funding of up to $900,000 over 3 years. In order to help build the 
next generation of research methodologists, NCER also began last year to provide an Early 
Career grant opportunity for applicants who are within 5 years of earning a PhD. Early 
Career grants offer up to $200,000 over 18 months and include a requirement that the 
researcher work with a more experienced mentor or advisor. 


Finally, NCER and NCSER support several Research Training Programs in Education 
Sciences to develop the next generation of researchers and to increase the field’s capacity to 
conduct high quality, scientific education research. Please refer to the accompanying 
document, “Research Training Programs in the Education Sciences,” for more details about 
these programs. 


Long-term Research Investment: 

NCER has invested $1.65 billion in its research and training programs since 2002, and 
NCSER has invested $541 million since 2006. The tables below provide a breakdown of the 
number of awards and funding levels for each Center’s competitions. As noted above, the 
Education Research Grants program and the Special Education Research Grants program 
are the largest, followed by the R&D Centers. 
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NCER Investments by Competition: 2002-2014 


Program Number of Awards Total Funding 
Education Research 636 $1.065B 
R&D Centers* 28 $333.1M 
Reseatcher-Practitioner 38 $75.7M 
Partnerships and 
Collaborations’ 

Small Business Innovation 86 $68.6M 
Reseatch 

Statistical & Research 45 $31.5M 
Methodology 

Unsolicited/Other 51 $78.7M 


NCSER Investments by Competition: 2006-2014 


Program Number of Awards Funding 
Special Education Research 247 $451.0M 
R&D Centers 7 $72.0M 
Postdoctoral Training/Early 15 $9.0M 


Career Development and 


Mentoring 


Small Business Innovation 11 $7.9M 
Research 
Unsolicited/Other 4 $1.7M 


* Includes the Reading for Understanding Research Initiative, a coordinated effort by IES and six 
research teams to develop and evaluate approaches to improve reading comprehension for all 


students. 


> Include Evaluation of State and Local Programs and Policies, which is now part of the Researcher- 
Practitioner Partnerships and Collaborations program but began several years earlier. 
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Topic Descriptions for Education Research Grants and Special Education Research 
Grants 


NCER Topics: 


Cognition and Student Learning — Research in this topic area focuses on theories of how 
the mind works to inform and improve education practice in reading, writing, mathematics, 
and science, and study skills from prekindergarten through grade 12. 


Early Learning Programs and Policies - NCER supports research that contributes to the 
improvement of school-readiness skills (e.g., pre-reading, language, vocabulary, early science 
and mathematics knowledge, social and behavioral competencies) of prekindergarten 
children (1e., 3- to 5-year-olds) in center-based settings. 


Education Technology — This topic area supports research that contributes to advances in 
technology intended to improve student achievement through the education system, 
including the development and evaluation of education technology products intended for 
use in schools or through formal programs run by schools or state and local education 
agencies. 


Effective Teachers and Effective Teaching — Research in this topic area focuses on 
instructional practices, programs, and strategies for improving the performance of classroom 
teachers in ways that promote student learning and academic achievement in reading, 
writing, mathematics, science, and — for English learners — English language proficiency, 
from kindergarten through grade 12. 


English Learners — NCER supports research in this topic area that contributes to the 
improvement of academic achievement in reading, writing, mathematics, or science for 
students in kindergarten through grade 12 who are English learners. 


Improving Education Systems: Policies, Organization, Management, and 
Leadership — The focus in this area is research that contributes to the improvement of the 
organization, programs, policies, and management of schools serving students in 
kindergarten through grade 12 and their state/local education agencies in order to help 
students succeed. 


Mathematics and Science Education — Research in this topic area contributes to the 
improvement of mathematics and science knowledge and skills of students from 
kindergarten through grade 12. 


Postsecondary and Adult Education — Research projects in this area are designed to 
contribute to the improvement of student education outcomes (a) at the college level (e., 
students working on certificates or associate’s or bachelor’s degrees) and (b) in adult 
education programs (i.e., students in adult English language programs or adult literacy 
programs including adult basic education, adult secondary education, and high school 
equivalency preparation programs). 


Mixed Methods in Education Research IES TWG | Meeting Summary 25 


Reading and Writing — NCER supports research that contributes to the improvement of 
learning, higher-order thinking, and achievement in reading and writing skills through the 
development and testing of tools and strategies (e.g., curricula, assessments, instructional 
approaches) for students from kindergarten through grade 12. 


Social and Behavioral Context for Academic Learning — Research in this topic area 
focuses on social skills, attitudes, and behaviors to improve or assess students’ 
social/behavioral competencies, and teacher practices that support them, that in the long- 
run will improve student academic achievement from kindergarten through grade 12. 


NCSER Topics: 

Autism Spectrum Disorders — This topic supports research on the development, 
implementation, and evaluation of comprehensive school-based interventions intended to improve 
multiple outcomes, including developmental, cognitive, communicative, academic, social, 
behavioral, and functional outcomes, for students identified with autism spectrum disorder 
from preschool through Grade 12. 


Cognition and Student Learning in Special Education — NCSER supports research on 
the principles of learning and information processing gained from cognitive science to 
improve developmental outcomes for infants and toddlers with disabilities or at risk for 
disabilities and learning for students with disabilities or at risk for disabilities in preschool 
through Grade 12. 


Early Intervention and Early Learning in Special Education - NCSER supports 
research that contributes to the improvement of developmental outcomes and school 
readiness of infants, toddlers, and young children (from birth through age 5) with disabilities 
ot at risk for disabilities. 


Families of Children with Disabilities — This topic supports research to identify effective 
strategies for improving family involvement in the education of their child with a disability 
and family support of their child with a disability in ways that ultimately improve educational 
or transition outcomes for students with disabilities from kindergarten through Grade 12. 


Mathematics and Science Education — This topic supports research that contributes to 
improvement of mathematics and science learning and achievement through the 
development and testing of tools and strategies for students with or at risk for disabilities 
from kindergarten through Grade 12. 


Professional Development for Teachers and Related Services Providers — The aim of 
this topic area is to identify effective strategies for improving the performance of current 
teachers, other instructional personnel, and related services providers in ways that increase 
reading, writing, language, mathematics, science, social, behavioral, or secondary transition 
outcomes, as well as functional skills that improve the educational outcomes of students 
with disabilities or at risk for disabilities from kindergarten through Grade 12. 


Reading, Writing, and Language Development — This topic supports research that 
contributes to the improvement of reading, writing, and language skills through the 
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development and testing of tools and strategies for students with or at risk for disabilities 
from kindergarten through Grade 12. 


Social and Behavioral Outcomes to Support Learning — NCSER supports research to 
contribute to the prevention or amelioration of behavior problems in students with or at risk 
for disabilities from kindergarten through grade 12 and concomitantly, improve their 
academic outcomes. 


Special Education Policy, Finance, and Systems - Through this program, NCSER 
supports research to improve outcomes for students with disabilities or at risk for disabilities 
from kindergarten through Grade 12 by identifying systemic processes, procedures, and 
programs that may be directly or indirectly linked to student outcomes. That is, rather than 
focusing on improving student outcomes by changing curricula or student-level intervention 
approaches, researchers conduct research on systems-level procedures and policies that are 
intended to improve the management, coordination, and implementation of systemic 
programs and services in ways that directly enhance the overall intervention or education 
environment and indirectly improve student outcomes. 


Technology for Special Education - NCSER supports research on education technology 
tools that are designed to improve outcomes for infants, toddlers, preschool children, and 
students from kindergarten through Grade 12 with or at risk for disabilities. 


Transition Outcomes for Special Education Secondary Students - Through this topic, 
NCSER supports research that contributes to the improvement of transition outcomes of 
secondary students with disabilities. Transition outcomes include the behavioral, social, 
communicative, functional, occupational, and academic skills that enable young adults with 
disabilities to obtain and hold meaningful employment, live independently, and obtain 
further training and education (e.g., postsecondary education, vocational education 


ptograms). 
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RESEARCH TRAINING PROGRAMS IN THE EDUCATION SCIENCES 


The research training programs run by the National Center for Education Research (NCER) 
and the National Center for Special Education Research (NCSER) are designed to prepare 
individuals to conduct rigorous and relevant education research that advances knowledge 
within the field and addresses issues important to education policymakers and practitioners. 
The training programs grew out of the findings and recommendations of the 2002 National 
Research Council report, Scientific Research in Education. Following an extensive review of the 
state of education research, the Council called for federal investment in “infrastructure- 
building programs to foster a scientifically competent, highly qualified community of 
education reseatchers.”° The Council’s recommendation reflected a concern that many 
education researchers were not sufficiently trained in statistics or causal inference. It also 
reflected a desire for education researchers to have multidisciplinary training and a solid 
grounding in ethics. 


Currently, NCER and NCSER provide funding for training programs targeting students in 
doctoral programs, post-doctoral students, early career researchers working in a higher 
education institution, and methods training for professionals in academic or applied research 
positions. Each of these is described below. 


e The Predoctoral Interdisciplinary Research Training Program in the Education 
Sciences (NCER) 


NCER launched the Predoctoral Interdisciplinary Research Training Program in the 
Education Sciences in 2004. It is a discretionary grant competition designed to establish 
doctoral research training programs in universities and provide fellowships to students. The 
training programs involve a number of academic fields, including economics, education, 
psychology, public policy, sociology, and statistics. Each training program has its own 
admission process and selection criteria for determining which students will recetve support. 
While fellowship length varies by training program, fellows generally receive 2 to 5 years of 
funding support that includes a $30,000 stipend, $10,500 for tuition and fees, and funds for 
teseatch and/or conference travel. ’ 


NCER has invested approximately $142 million in the predoctoral training program since it 
began. A total of 18 institutions have received at least one grant. Each training program is 
funded for 5 years, but all programs can request 1 or 2 year no-cost extensions and can apply 
to renew their awards for an additional 5 years.” As of FY 2014, a total of 31 predoctoral 
training grants have been awarded, including 8 universities with 1 grant; 7 universities with 2 
grants; and 3 universities with 3 grants. The programs have trained over 700 students. 


e Postdoctoral Research Training Programs (NCER and NCSER) 


6 National Research Council (2002). Scentific Research in Education. Washington, D.C.: National 
Academy Press. Page 9. 

7In FY 2014, the maximum predoctoral fellowship length was reduced from 5 years to 4 years. 

8 The predoctoral training programs funded in FY 2004/2005 were able to apply for renewal in FY 
2008/2009 and again for a second time in FY 2014. Training programs originally funded in FY 
2008/2009 were eligible for renewal for the first time in FY 2014. There is an additional FY15 
competition that provides an opportunity for the remaining programs to renew. 
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NCER and NCSER established the Postdoctoral Research Training Program in 2005 and 
2008, respectively. These programs were designed to prepare education researchers to 
conduct high-quality, independent education research that advances knowledge within the 
field of education sciences and addresses issues important to education leaders and 
practitioners. Grants are awarded to academic institutions that award doctoral degrees in 
fields relevant to education. An institution may hold more than one Postdoctoral award, 
provided that there is no overlap of Project Directors. In most instances, the Project 
Directors are senior researchers who have been funded by IES or other funders to lead one 
or more studies. Different models of fellowships are permitted. For example, an individual 
faculty member may submit a proposal to train one or two fellows, or several faculty 
members may propose to jointly train a few fellows over the grant period. Likewise, the 
number of participating fellows and the length of the fellowship can vary. Grantees typically 
propose to prepare 4 fellows for 2 to 3 years per fellowship during the award period. 
Currently, the maximum length of the award period is 5 years, and the maximum award 
amount is $700,000. 


Since 2005, NCER has invested over $24 million in 39 Postdoctoral Training Research 
Grants. As with the predoctoral training awards, several institutions have received two or 
three awards. To date, 114 fellows have completed their NCER-funded postdoctoral 
training. NCER did not run a competition for postdoctoral training programs in FY 2014 
because of funding constraints, but is once again running a competition in FY 2015. 


Since 2008, the NCSER has committed nearly $8 million to support 12 Special Education 
Postdoctoral Training Research Grants. Eleven universities have recetved one grant, and one 
university has received two grants. A total of 34 fellows have completed their NCSER- 
funded postdoctoral training. Due to funding constraints, NCSER’s postdoctoral training 
program has not been competed since 2012. 


e Research Training Program in Special Education: Early Career Development and 
Mentoring (NCSER) 

NCSER initiated this new training program in 2013. The program is designed to help early 
career scientists further develop their research skills and begin their own line of rigorous 
reseatch in early intervention or special education. Early Career grants provide support for 
research and career development that includes training under the guidance of an experienced 
mentor or mentors. The program is intended to launch independent research careers 
focused on students with or at risk for disabilities and to prepare researchers to conduct the 
type of research that IES funds. The Early Career scientist serves as the Principal 
Investigator for the grant. NCSER funded three early career grants in 2013 and has an Early 
Career competition in 2015. For 2015, the early career scientists must (1) have completed 
their doctoral degree or postdoctoral program no earlier than May 1, 2012, (2) hold a tenure- 
track or research faculty position at an institution of higher education, and (3) not have 
previously served as a PI or Co-PI on an IES research grant or been a postdoctoral fellow 
through an IES postdoctoral training program. Early Career grants extend a maximum of 4 
years and provide up to $400,000. 
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Methods Training in the Education Sciences (NCER and NCSER) 


Since 2007, NCER and NCSER have sought to increase the overall capacity of the field to 
conduct rigorous and relevant education research by awarding Methods Training grants to 
help current education researchers maintain and upgrade their research and analysis skills. 
Unlike the programs described above, the Methods Training program often targets mid- 
cateer researchers. The primary mechanism has been a series of 17 summer research training 
institutes hosted by research universities (typically 1-2 weeks) that focus on how to conduct 
specific types of education research studies: cluster-randomized trials (8 trainings), single- 
case intervention studies (5), quasi-experiments (3), and within-study comparisons (1). These 
summer institutes have been funded through the Institute’s unsolicited grants program and 
through contracts. 


NCER and NCSER have awarded nearly $4 million in grants and contracts to support 
summer institutes on research methodology since 2008. The most recent Methods Training 
awards were to continue the summer institutes on cluster-randomized trials and quasi- 
experiments, and to create a new training institute on cost-benefit and cost-effectiveness 
analyses. For FY15, NCER encouraged applications that addressed the following: 


¢ The formation and analysis of longitudinal data sets based on state or district 
administrative data; 

¢ Collection and analysis of data to explain the variation in impacts across sites in 
large-scale experimental and quasi-experimental studies; 

¢ Iterative processes used in the development, implementation, testing, and 
improvement of education interventions; and 

¢ Involvement of education practitioners and policymakers in the design and 
implementation of research. 


At this time, NCSER does not have a separate competition for research methods in special 
education, though special education researchers may apply for and participate in any of the 
training programs funded through NCER. 
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